Repository navigation
feat(review): opt-in AI input coverage, hunk excerpts and a dbt manifest context card - #1771
RaghwendraSingh wants to merge 69 commits into
Conversation
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the
|
There was a problem hiding this comment.
13 issues found across 9 files
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="packages/opencode/test/altimate/fixtures/review-ai-prompt-input.ts">
<violation number="1" location="packages/opencode/test/altimate/fixtures/review-ai-prompt-input.ts:12">
P3: The hunk header counts in `bigDiff` don't match the hunk body: the old side has 3 lines and the new side has `size + 2`, but the header says `size + 2` and `size + 4`. Use the real counts so this fixture is a valid unified diff for any future hunk-window test.</violation>
</file>
<file name="packages/opencode/src/altimate/review/ai-review.ts">
<violation number="1" location="packages/opencode/src/altimate/review/ai-review.ts:391">
P2: A diff between `coverage.diffChars` and `maxChars` bypasses `excerptDiff`, so a lower `ALTIMATE_REVIEW_AI_COVERAGE_DIFF_CHARS` override is ignored. Compare against the effective limit, such as `Math.min(maxChars, coverage.diffChars)`.</violation>
<violation number="2" location="packages/opencode/src/altimate/review/ai-review.ts:421">
P3: Batch sizing counts every card, but `buildBatchMessage` omits cards after the 6,000-character per-call cap. Card-heavy inputs can therefore split into unnecessary model calls; apply the same cumulative card cap when estimating batches.</violation>
<violation number="3" location="packages/opencode/src/altimate/review/ai-review.ts:649">
P2: `planAiInput` now runs outside the `try` in `runAiReview`, so an exception while rendering the prompt escapes the lane. The doc comment promises this function never throws. Move `const plan = planAiInput(input)` inside the `try`, or wrap the planning in its own guard that returns an `error` status.</violation>
</file>
<file name="packages/opencode/src/altimate/review/runner.ts">
<violation number="1" location="packages/opencode/src/altimate/review/runner.ts:570">
P2: Column tests disappear from the card for manifests without `attached_node`; fall back to `depends_on.nodes` and `test_metadata.kwargs.column_name` when those fields are absent.</violation>
<violation number="2" location="packages/opencode/src/altimate/review/runner.ts:603">
P3: `text()` keeps only `partition_by.field` and drops `granularity`. Include the granularity in the rendered value, since the card is meant to show partitioning facts the diff does not.</violation>
<violation number="3" location="packages/opencode/src/altimate/review/runner.ts:627">
P2: `macroCard` tries the name before the changed file's path. For a file like macros/sf/hash.sql, the card then describes the dispatcher `hash` from another file, not the macro this PR changed. Try an exact `original_file_path` match first, then fall back to name.</violation>
<violation number="4" location="packages/opencode/src/altimate/review/runner.ts:639">
P2: These `.slice()` caps run before rendering, so the card's `+N more` marker can never show for callers, columns or siblings. A macro with 40 callers reads as having 10. Return the full lists and let `renderContextCard`'s `list()` cap them, or keep the true totals.</violation>
</file>
<file name="packages/opencode/src/altimate/review/ai-context-card.ts">
<violation number="1" location="packages/opencode/src/altimate/review/ai-context-card.ts:95">
P2: Seed files never get a context card. `MODEL_KINDS` leaves out `seed`, although `ModelCard` and `modelCard()` are documented to cover seeds. Add `seed` to the set, and drop the `kind === "model"` filter in the yml/test branch if seed parents should be carded there too.</violation>
<violation number="2" location="packages/opencode/src/altimate/review/ai-context-card.ts:122">
P3: The comment says a yml card covers 'the models it declares or tests', but `graphParents([f.path])` only returns models that generic tests in that yml depend on. A yml that declares a model without tests gets no card. Either fix the comment to say 'tests', or resolve declared models separately.</violation>
<violation number="3" location="packages/opencode/src/altimate/review/ai-context-card.ts:123">
P2: Schema and test changes attached to seed or snapshot nodes lose their related context card because this filter accepts only models. Include the resource kinds supported by `modelCard`.</violation>
</file>
<file name="packages/opencode/src/altimate/review/orchestrate.ts">
<violation number="1" location="packages/opencode/src/altimate/review/orchestrate.ts:2477">
P3: Seed files never receive the context card this new interface promises, because `attachContextCards` excludes `seed` from its cardable kinds. Add `seed` to the helper's model-card kinds.</violation>
<violation number="2" location="packages/opencode/src/altimate/review/orchestrate.ts:2549">
P2: These quota-skipped files were still checked by the deterministic lanes, but the ledger shows them under "Not reviewed". Consider a separate section or label (for example "Not sent to AI") so the header does not overstate coverage.</violation>
</file>
Reply with feedback, questions, or to request a fix.
View guided diff | Turn on auto-fix | Re-trigger cubic
| const replay = aiReplayPath() | ||
| if (replay) return runAiReplay(input, replay) | ||
| const startedAt = Date.now() | ||
| const plan = planAiInput(input) |
There was a problem hiding this comment.
P2: planAiInput now runs outside the try in runAiReview, so an exception while rendering the prompt escapes the lane. The doc comment promises this function never throws. Move const plan = planAiInput(input) inside the try, or wrap the planning in its own guard that returns an error status.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/opencode/src/altimate/review/ai-review.ts, line 649:
<comment>`planAiInput` now runs outside the `try` in `runAiReview`, so an exception while rendering the prompt escapes the lane. The doc comment promises this function never throws. Move `const plan = planAiInput(input)` inside the `try`, or wrap the planning in its own guard that returns an `error` status.</comment>
<file context>
@@ -263,13 +646,20 @@ export async function runAiReview(input: AiReviewInput): Promise<AiReviewResult>
const replay = aiReplayPath()
if (replay) return runAiReplay(input, replay)
const startedAt = Date.now()
+ const plan = planAiInput(input)
+ const { files, budget } = plan
+ const dryRun = aiDryRunEnabled()
</file context>
There was a problem hiding this comment.
Not changing: the only caller wraps the lane in try/catch and records state: "error" (orchestrate.ts ~2498), so nothing escapes. Only the "never throws" doc wording is loose.
| ...(manifestMissing && !anyManifest | ||
| ? modelFiles.map((f) => ({ path: f.path, reason: "no dbt manifest: lineage, config and graph checks skipped" })) | ||
| : []), | ||
| ...aiUnsent.map((p) => ({ path: p, reason: "not sent to the AI (file quota)" })), |
There was a problem hiding this comment.
P2: These quota-skipped files were still checked by the deterministic lanes, but the ledger shows them under "Not reviewed". Consider a separate section or label (for example "Not sent to AI") so the header does not overstate coverage.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/opencode/src/altimate/review/orchestrate.ts, line 2549:
<comment>These quota-skipped files were still checked by the deterministic lanes, but the ledger shows them under "Not reviewed". Consider a separate section or label (for example "Not sent to AI") so the header does not overstate coverage.</comment>
<file context>
@@ -2532,6 +2546,7 @@ export async function runReview(input: OrchestrateInput): Promise<VerdictEnvelop
...(manifestMissing && !anyManifest
? modelFiles.map((f) => ({ path: f.path, reason: "no dbt manifest: lineage, config and graph checks skipped" }))
: []),
+ ...aiUnsent.map((p) => ({ path: p, reason: "not sent to the AI (file quota)" })),
]
const seen = new Set<string>()
</file context>
There was a problem hiding this comment.
Follow-up: low priority, tracked. The reason text is right; we'll move these files to a separate "not sent to AI" group so the count is right too.
| const byName = macros.filter(([, m]) => m?.name === name) | ||
| const path = cardOpts?.path | ||
| const hit = | ||
| byName.find(([, m]) => m.package_name === project) ?? |
There was a problem hiding this comment.
P2: macroCard tries the name before the changed file's path. For a file like macros/sf/hash.sql, the card then describes the dispatcher hash from another file, not the macro this PR changed. Try an exact original_file_path match first, then fall back to name.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/opencode/src/altimate/review/runner.ts, line 627:
<comment>`macroCard` tries the name before the changed file's path. For a file like macros/sf/hash.sql, the card then describes the dispatcher `hash` from another file, not the macro this PR changed. Try an exact `original_file_path` match first, then fall back to name.</comment>
<file context>
@@ -531,6 +532,129 @@ export function createDispatcherRunner(opts: DispatcherRunnerOptions): ReviewRun
+ const byName = macros.filter(([, m]) => m?.name === name)
+ const path = cardOpts?.path
+ const hit =
+ byName.find(([, m]) => m.package_name === project) ??
+ byName[0] ??
+ (path
</file context>
There was a problem hiding this comment.
Follow-up: low priority, tracked. Matching original_file_path first is the right order; it only matters for dispatcher macros that share a basename.
| .filter(([id, n]) => id.startsWith("model.") && asArray<string>(n?.depends_on?.macros).includes(uid)) | ||
| .map(([, n]) => String(n.name)) | ||
| .sort((a, b) => Number(changed.has(b)) - Number(changed.has(a))) | ||
| .slice(0, 10) |
There was a problem hiding this comment.
P2: These .slice() caps run before rendering, so the card's +N more marker can never show for callers, columns or siblings. A macro with 40 callers reads as having 10. Return the full lists and let renderContextCard's list() cap them, or keep the true totals.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/opencode/src/altimate/review/runner.ts, line 639:
<comment>These `.slice()` caps run before rendering, so the card's `+N more` marker can never show for callers, columns or siblings. A macro with 40 callers reads as having 10. Return the full lists and let `renderContextCard`'s `list()` cap them, or keep the true totals.</comment>
<file context>
@@ -531,6 +532,129 @@ export function createDispatcherRunner(opts: DispatcherRunnerOptions): ReviewRun
+ .filter(([id, n]) => id.startsWith("model.") && asArray<string>(n?.depends_on?.macros).includes(uid))
+ .map(([, n]) => String(n.name))
+ .sort((a, b) => Number(changed.has(b)) - Number(changed.has(a)))
+ .slice(0, 10)
+ const macroName = String(macro.name)
+ const sep = macroName.indexOf("__")
</file context>
There was a problem hiding this comment.
Follow-up: low priority, tracked. We'll pass the totals so the card can say how many it left out.
| return text.slice(0, maxChars - 1) + "…" | ||
| } | ||
|
|
||
| const MODEL_KINDS = new Set(["model_sql", "python_model", "snapshot"]) |
There was a problem hiding this comment.
P2: Seed files never get a context card. MODEL_KINDS leaves out seed, although ModelCard and modelCard() are documented to cover seeds. Add seed to the set, and drop the kind === "model" filter in the yml/test branch if seed parents should be carded there too.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/opencode/src/altimate/review/ai-context-card.ts, line 95:
<comment>Seed files never get a context card. `MODEL_KINDS` leaves out `seed`, although `ModelCard` and `modelCard()` are documented to cover seeds. Add `seed` to the set, and drop the `kind === "model"` filter in the yml/test branch if seed parents should be carded there too.</comment>
<file context>
@@ -0,0 +1,134 @@
+ return text.slice(0, maxChars - 1) + "…"
+}
+
+const MODEL_KINDS = new Set(["model_sql", "python_model", "snapshot"])
+const DECLARING_KINDS = new Set(["schema_yml", "source_yml", "test"])
+
</file context>
| const MODEL_KINDS = new Set(["model_sql", "python_model", "snapshot"]) | |
| const MODEL_KINDS = new Set(["model_sql", "python_model", "snapshot", "seed"]) |
There was a problem hiding this comment.
Follow-up: low priority, tracked. We'll either add seed or drop the unreachable seed lookup.
| const out: string[] = [] | ||
| for (let h = 0; h < hunks; h++) { | ||
| const start = 10 + h * 100 | ||
| out.push(`@@ -${start},${size + 2} +${start},${size + 4} @@`) |
There was a problem hiding this comment.
P3: The hunk header counts in bigDiff don't match the hunk body: the old side has 3 lines and the new side has size + 2, but the header says size + 2 and size + 4. Use the real counts so this fixture is a valid unified diff for any future hunk-window test.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/opencode/test/altimate/fixtures/review-ai-prompt-input.ts, line 12:
<comment>The hunk header counts in `bigDiff` don't match the hunk body: the old side has 3 lines and the new side has `size + 2`, but the header says `size + 2` and `size + 4`. Use the real counts so this fixture is a valid unified diff for any future hunk-window test.</comment>
<file context>
@@ -0,0 +1,64 @@
+ const out: string[] = []
+ for (let h = 0; h < hunks; h++) {
+ const start = 10 + h * 100
+ out.push(`@@ -${start},${size + 2} +${start},${size + 4} @@`)
+ out.push(` select ${h} as ctx_before`)
+ for (let i = 0; i < size; i++) out.push(`+ , coalesce(col_${h}_${i}, 'unknown_value_${i}') as renamed_column_${h}_${i}`)
</file context>
| out.push(`@@ -${start},${size + 2} +${start},${size + 4} @@`) | |
| out.push(`@@ -${start},3 +${start},${size + 2} @@`) |
There was a problem hiding this comment.
Not changing: the parser doesn't use the counts, and the fixture feeds a byte-identical snapshot that has to stay as it is.
| materialized: text(config.materialized), | ||
| uniqueKey: text(config.unique_key), | ||
| incrementalStrategy: text(config.incremental_strategy), | ||
| partitionBy: text(config.partition_by), |
There was a problem hiding this comment.
P3: text() keeps only partition_by.field and drops granularity. Include the granularity in the rendered value, since the card is meant to show partitioning facts the diff does not.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/opencode/src/altimate/review/runner.ts, line 603:
<comment>`text()` keeps only `partition_by.field` and drops `granularity`. Include the granularity in the rendered value, since the card is meant to show partitioning facts the diff does not.</comment>
<file context>
@@ -531,6 +532,129 @@ export function createDispatcherRunner(opts: DispatcherRunnerOptions): ReviewRun
+ materialized: text(config.materialized),
+ uniqueKey: text(config.unique_key),
+ incrementalStrategy: text(config.incremental_strategy),
+ partitionBy: text(config.partition_by),
+ clusterBy: text(config.cluster_by),
+ contractEnforced: config.contract?.enforced === true || node.contract?.enforced === true,
</file context>
There was a problem hiding this comment.
Not changing for now: the card shows the partition column. Granularity is a nice-to-have, and we may add it with the next card revision.
| return macro ? [{ name: macro.name, macro }] : [] | ||
| } | ||
| if (!DECLARING_KINDS.has(kind)) return [] | ||
| const parents = (await runner.graphParents?.([f.path])) ?? [] |
There was a problem hiding this comment.
P3: The comment says a yml card covers 'the models it declares or tests', but graphParents([f.path]) only returns models that generic tests in that yml depend on. A yml that declares a model without tests gets no card. Either fix the comment to say 'tests', or resolve declared models separately.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/opencode/src/altimate/review/ai-context-card.ts, line 122:
<comment>The comment says a yml card covers 'the models it declares or tests', but `graphParents([f.path])` only returns models that generic tests in that yml depend on. A yml that declares a model without tests gets no card. Either fix the comment to say 'tests', or resolve declared models separately.</comment>
<file context>
@@ -0,0 +1,134 @@
+ return macro ? [{ name: macro.name, macro }] : []
+ }
+ if (!DECLARING_KINDS.has(kind)) return []
+ const parents = (await runner.graphParents?.([f.path])) ?? []
+ const names = [...new Set(parents.filter((p) => p.kind === "model").map((p) => p.name))].slice(0, 3)
+ const cards = await Promise.all(names.map(async (name) => ({ name, model: await runner.modelCard?.(name) })))
</file context>
There was a problem hiding this comment.
Fair nit; we'll fix the wording the next time this file is touched.
| aiFiles.push({ path: f.path, status: f.status, model: modelNameFromPath(f.path), kind: f.kind, diff: f.diff, sql: contents[i] }), | ||
| ) | ||
| touch(aiFiles) | ||
| if (input.aiContextCard && anyManifest) await attachContextCards(aiFiles, input.runner) |
There was a problem hiding this comment.
P3: Seed files never receive the context card this new interface promises, because attachContextCards excludes seed from its cardable kinds. Add seed to the helper's model-card kinds.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/opencode/src/altimate/review/orchestrate.ts, line 2477:
<comment>Seed files never receive the context card this new interface promises, because `attachContextCards` excludes `seed` from its cardable kinds. Add `seed` to the helper's model-card kinds.</comment>
<file context>
@@ -2466,6 +2474,7 @@ export async function runReview(input: OrchestrateInput): Promise<VerdictEnvelop
aiFiles.push({ path: f.path, status: f.status, model: modelNameFromPath(f.path), kind: f.kind, diff: f.diff, sql: contents[i] }),
)
touch(aiFiles)
+ if (input.aiContextCard && anyManifest) await attachContextCards(aiFiles, input.runner)
const context = input.aiGraphContext ? await aiGraphContext(aiFiles.map((f) => f.path), input.runner, getContent) : undefined
// The verifier reads engine SQL first (the same SQL the engine lanes saw), then compiled output.
</file context>
There was a problem hiding this comment.
Duplicate of r4626982915; tracked there.
| let size = 0 | ||
| for (const f of files) { | ||
| const n = fileSection(f, budget.maxFileChars).length | ||
| const n = fileSection(f, budget.maxFileChars, budget.coverage).length |
There was a problem hiding this comment.
P3: Batch sizing counts every card, but buildBatchMessage omits cards after the 6,000-character per-call cap. Card-heavy inputs can therefore split into unnecessary model calls; apply the same cumulative card cap when estimating batches.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/opencode/src/altimate/review/ai-review.ts, line 421:
<comment>Batch sizing counts every card, but `buildBatchMessage` omits cards after the 6,000-character per-call cap. Card-heavy inputs can therefore split into unnecessary model calls; apply the same cumulative card cap when estimating batches.</comment>
<file context>
@@ -105,7 +418,7 @@ export function batchAiFiles(files: AiReviewFile[], budget: AiInputBudget): AiRe
let size = 0
for (const f of files) {
- const n = fileSection(f, budget.maxFileChars).length
+ const n = fileSection(f, budget.maxFileChars, budget.coverage).length
if (cur.length && size + n > budget.batchChars) {
batches.push(cur)
</file context>
There was a problem hiding this comment.
Follow-up: low priority, tracked. Sizing over-counts cards, so batches come out smaller than needed, never larger.
| /** Head-side lines of context kept around each changed line in a file excerpt. */ | ||
| const EXCERPT_CONTEXT_LINES = 20 | ||
| /** Characters of a file's diff section under coverage (hunks beyond it are listed as omitted). */ | ||
| const COVERAGE_DIFF_CHARS = 16_000 |
There was a problem hiding this comment.
WARNING: Coverage mode can build a single ~660k-char call with no per-call cap
With ALTIMATE_REVIEW_AI_COVERAGE=1 the diff section can be COVERAGE_DIFF_CHARS (16,000) per file plus up to 6,000 of current file, while the new quotas allow 30 files (12 models + 8 macros + 4 tests + 6 yml) and batchChars still defaults to Infinity (line 57), so batchAiFiles keeps them all in one batch. One call can reach ~30 x 22k = 660k chars (~165k tokens) versus <= 20 x 12k = 240k before. That overruns a 128k-token model's context, reviewBatch throws, every batch fails and the lane reports error with no findings - on exactly the large PRs this mode targets. Consider defaulting batchChars to a bounded value when coverage is on, or scaling the per-file budgets against a per-call total.
There was a problem hiding this comment.
Fixed in bf6e861: with ALTIMATE_REVIEW_AI_COVERAGE=1, batchChars now defaults to 120k chars (about 30k tokens), so 30 quota files split across calls instead of one ~660k-char call. ALTIMATE_REVIEW_AI_BATCH_CHARS still overrides it, and with coverage off the default stays Infinity (the prompt snapshot test is unchanged).
| const BUCKET_ORDER = [...new Set(KIND_ORDER.map((k) => BUCKET_OF[k]))] | ||
|
|
||
| /** Added and removed lines of a unified diff (file headers before the first hunk excluded). */ | ||
| export function changedLineCount(diff: string | undefined): number { |
There was a problem hiding this comment.
SUGGESTION: Duplicates the existing countChangedLines
diff-filter.ts:329 already exports countChangedLines (imported from the same module for walkDiffLines) for this exact job, and it correctly ignores +++/--- file headers. This copy counts +++ b/... and --- a/... as changed lines whenever the diff has no @@ hunk (first === -1, so Math.max(first, 0) === 0), which inflates the most-changed ordering used by selectAiFilesByQuota for header-less diffs. Reuse countChangedLines(f.diff) here instead.
There was a problem hiding this comment.
Fair point; we'll reuse countChangedLines in a follow-up cleanup. Behaviour is the same for git diffs, which always carry @@ headers.
| if (!kept.length) return { text: truncate(content, maxChars), truncated: true } | ||
| // Fill what is left with the file head, before the first window. | ||
| let head = 0 | ||
| while (head + 1 < kept[0][0] && used + size(1, head + 1) + MARKER <= maxChars) head++ |
There was a problem hiding this comment.
SUGGESTION: Quadratic head fill
size(1, head + 1) re-slices and re-sums lines 1..head+1 on every iteration, so extending the head over H lines costs O(H^2) with a fresh array each step (H can approach maxChars for files with many short or blank lines, and this runs once per file under coverage). Accumulate the running length instead, e.g. keep headChars and add lines[head].length + 1 per step, then test used + headChars + MARKER <= maxChars.
There was a problem hiding this comment.
Not changing now: it's bounded by the per-file char budget (a few hundred lines). We'll switch it to a running sum if profiling ever shows it.
| return text.slice(0, maxChars - 1) + "…" | ||
| } | ||
|
|
||
| const MODEL_KINDS = new Set(["model_sql", "python_model", "snapshot"]) |
There was a problem hiding this comment.
SUGGESTION: Changed seeds never get a context card
MODEL_KINDS omits "seed" and seeds are not in DECLARING_KINDS, so cardsFor returns [] for every seed file. But runner.modelCard explicitly resolves seed. nodes (runner.ts:543) and ModelCard is documented as facts about "one model, snapshot or seed", so that branch is unreachable through this feature. Either add "seed" to MODEL_KINDS (a changed seed is a legitimate review target) or drop the seed lookup from modelCard.
There was a problem hiding this comment.
Same root cause as r4626982915; tracked there.
Code Review SummaryStatus: 3 Issues Found | Recommendation: Address before merge Overview
Issue Details (click to expand)SUGGESTION
Files Reviewed (9 files)
Note: the per-call size bound raised earlier is now fixed ( Fix these issues in Kilo Cloud Previous Review Summary (commit b07b80d)Current summary above is authoritative. Previous snapshots are kept for context only. Previous review (commit b07b80d)Status: 4 Issues Found | Recommendation: Address before merge Overview
Issue Details (click to expand)WARNING
SUGGESTION
Note: the PR description omits the repository's required "Cost side effects" section (the automated issue-compliance check already flagged this). The lane is opt-in and bounded by file quotas and character budgets, but it does raise per-review token spend ~1.25x when enabled, so that section should be filled in. Fix these issues in Kilo Cloud Files Reviewed (9 files)
Reviewed by deepseek-v4.1-flash-1017 · Input: 173.1K · Output: 11.4K · Cached: 578.9K Review guidance: REVIEW.md from base branch |
hkaltai
left a comment
There was a problem hiding this comment.
Verdict: approve, with minor follow-ups. Panel: Claude, Codex and lead reviewer, covering all 9 changed files.
The core promise holds: with the flags unset the prompt is byte-identical to the base (confirmed independently by two reviewers against the snapshot), and the 31 tests in review-ai-inputs.test.ts pass. All findings concern behavior behind the opt-in flags and advisory context quality, so none is CRITICAL or MAJOR.
Counts: CRITICAL 0, MAJOR 0, MINOR 8 (inline), NIT 10.
Most worthwhile follow-ups: bound the coverage-mode call size (batchChars default), make compiled-SQL windowing robust when added lines are Jinja-only, and resolve model/macro cards by project and path rather than by first name match.
Nits
ai-review.ts:51numrequiresn > 0:ALTIMATE_REVIEW_AI_QUOTA_TEST=0silently becomes 4, so a bucket cannot be disabled. Quoted:return v && Number.isFinite(n) && n > 0 ? n : d. Fix: separate parser allowing 0 for quotas.excerptFilewindow excerpt (ai-review.ts:327area): when the first context window exceeds the budget, the beginning is kept and the changed anchor can be dropped (350-char lines, change at line 100, 6,000 budget). Fix: reserve space for changed lines first, then expand context.runner.ts:640-649: a macro name containing__splits at the first__, so an unrelatedget_x__yis listed as a variant ofy. Fix: only treatdefault/adapter prefixes or dispatchers as variants.ai-review.ts:240: preamble longer than budget emits an empty "(hunk truncated)" but reportshunksSent: 1. Fix: record 0 whenmaxChars - used <= 0.ai-review.ts:313:excerptFilecan exceedmaxCharsby about 40 chars (trailing marker not reserved when no head window). Fix: reserve it.ai-context-card.tsMODEL_KINDSlacksseedalthough theModelCarddoc says "model, snapshot or seed". Fix: add seed or fix the doc.- Comment at
ai-review.ts:563says cards stop once the call holds 6,000 chars, but a too-large card is skipped and later smaller ones are still added. Reword. planAiInputdoc says "Pure apart from readingenv", butenvis a parameter.- Naming:
AiReviewFile.card(string) vsAiInputRecord.card(boolean);fileCharsinrenderFileis a budget (fileBudget);CoverageArg = boolean | AiCoverageRender | undefinedis a test-only tri-state; localconst MARKER = 40could be module-levelMARKER_RESERVE. - Missing tests: dry-run+replay+verify, duplicate model names across packages, macro basename/path collisions, YAML cards via the real runner (not a mocked
graphParents), Jinja-only changes on compiled SQL, bounded size under coverage, and rendering of quota-unsent files in the "Not reviewed" block.
| @@ -263,13 +646,20 @@ export async function runAiReview(input: AiReviewInput): Promise<AiReviewResult> | |||
| const replay = aiReplayPath() | |||
| if (replay) return runAiReplay(input, replay) | |||
There was a problem hiding this comment.
[MINOR] Replay returns before the dry-run check
if (replay) return runAiReplay(input, replay)
With ALTIMATE_REVIEW_AI_DRYRUN=1, ALTIMATE_REVIEW_AI_REPLAY=<envelope> and ALTIMATE_REVIEW_AI_VERIFY=1, replay returns before dry-run is evaluated, so a stored envelope with AI findings still reaches the verifier (a model call) and no ai.input is recorded. Replay is an eval-only path, hence MINOR, but it breaks the no-model-call promise of dry-run. Fix: evaluate dry-run (and plan the input) before dispatching replay, and add a regression test. (Codex)
| * lines (8+ non-space chars, no Jinja) are found by text, and "unlocated" otherwise. | ||
| */ | ||
| function locateAddedLines(lines: string[], rows: Array<{ line: number; text: string; kind: string }>): number[] | "unlocated" | undefined { | ||
| const eligible = rows.filter((r) => r.kind === "+" && !JINJA.test(r.text) && r.text.replace(/\s/g, "").length >= 8) |
There was a problem hiding this comment.
[MINOR] Compiled-SQL windows fall back to raw line numbers when no added line is eligible
const eligible = rows.filter((r) => r.kind === "+" && !JINJA.test(r.text) && r.text.replace(/\s/g, "").length >= 8)
When every added line is Jinja or under 8 non-space chars ({{ ref(..) }} swaps, {% if is_incremental() %}, , id), eligible is empty and the function returns undefined, which excerptFile treats as "raw line numbers match". On compiled SQL where macro expansion shifted lines, windows are then centred on the wrong lines and the unapproximated marker is not emitted. The diff itself is still sent, so this degrades excerpt quality only (opt-in flag). Fix: let the caller say whether f.sql is compiled, or also check context rows; return "unlocated" if nothing can be verified. Add a Jinja-only test on compiled SQL. (Claude)
| maxFiles: num(env.ALTIMATE_REVIEW_AI_MAX_FILES, MAX_FILES), | ||
| maxFileChars: num(env.ALTIMATE_REVIEW_AI_MAX_FILE_CHARS, MAX_DIFF_CHARS), | ||
| batchChars: num(env.ALTIMATE_REVIEW_AI_BATCH_CHARS, Infinity), | ||
| coverage: |
There was a problem hiding this comment.
[MINOR] Coverage mode enlarges the single call with no bound
batchChars: num(env.ALTIMATE_REVIEW_AI_BATCH_CHARS, Infinity),
Coverage raises the file cap to 30 and adds a per-diff budget, while batchChars stays Infinity. A 40-large-model PR was measured at about 242k chars by default vs about 658k with coverage, which can exceed context or the 300 s timeout and return no AI findings at all. The unbounded single call is pre-existing, but coverage makes it markedly worse. Fix: default batchChars to a finite value when coverage is on, or cap total chars and push overflow into unsent. (Claude)
| const nodes = Object.entries<any>(raw.nodes ?? {}) | ||
| const isModel = (id: string) => id.startsWith("model.") | ||
| const hit = | ||
| nodes.find(([id, n]) => isModel(id) && n?.name === name) ?? |
There was a problem hiding this comment.
[MINOR] modelCard picks the first same-named model across packages
nodes.find(([id, n]) => isModel(id) && n?.name === name) ??
A dependency package model with the same name listed earlier in the manifest supplies materialization, keys, columns and graph facts for the changed project model (reproduced by both external reviewers). The card is presented as authoritative. Fix: prefer metadata.project_name (as macroCard does) and/or match by path or unique id. (Codex, Claude)
| const byName = macros.filter(([, m]) => m?.name === name) | ||
| const path = cardOpts?.path | ||
| const hit = | ||
| byName.find(([, m]) => m.package_name === project) ?? |
There was a problem hiding this comment.
[MINOR] macroCard resolves by file basename before path, and not within the project
byName.find(([, m]) => m.package_name === project) ??
byName[0] ??
The lookup name is the file basename, not necessarily the macro defined in it. macros/helpers.sql defining foo can get the card of an unrelated helpers macro, or a package macro (e.g. dbt_utils.date_spine) with its callers and variants. The path fallback is also not restricted to the project package. Fix: match by path within the project first, then name within the project, then any package. (Codex, Claude)
| return macro ? [{ name: macro.name, macro }] : [] | ||
| } | ||
| if (!DECLARING_KINDS.has(kind)) return [] | ||
| const parents = (await runner.graphParents?.([f.path])) ?? [] |
There was a problem hiding this comment.
[MINOR] YAML declaration cards depend on test-node parents, not patch_path
const parents = (await runner.graphParents?.([f.path])) ?? []
graphParents links a YAML file to nodes via tests, while models are tied to their YAML by patch_path. A schema YAML with no generic tests gets no card even though the manifest has its models' config; removing the last test removes the card. The orchestrator test mocks graphParents, which hides this. Fix: resolve declarations through patch_path. (Codex, Claude)
| : fileChars < maxChars | ||
| ? truncate(f.sql, fileChars) | ||
| : excerptFile(f.sql, f.diff, maxChars).text | ||
| if (f.sql.length > fileChars) stats = { ...stats, truncated: true } |
There was a problem hiding this comment.
[MINOR] Ledger reports truncated:false when MAX_FILE_CHARS < 2000
if (f.sql.length > fileChars) stats = { ...stats, truncated: true }
With ALTIMATE_REVIEW_AI_MAX_FILE_CHARS=1000 and 1,500-char SQL, fileChars is widened to 2000 but excerptFile(..., maxChars) still truncates, so the record says truncated:false while the prompt contains (truncated). Fix: compare against the budget actually applied (Math.min(maxChars, ...)). (Claude)
| ...(manifestMissing && !anyManifest | ||
| ? modelFiles.map((f) => ({ path: f.path, reason: "no dbt manifest: lineage, config and graph checks skipped" })) | ||
| : []), | ||
| ...aiUnsent.map((p) => ({ path: p, reason: "not sent to the AI (file quota)" })), |
There was a problem hiding this comment.
[MINOR] Quota-omitted files listed under Not reviewed although engine lanes checked them
...aiUnsent.map((p) => ({ path: p, reason: "not sent to the AI (file quota)" })),
The ledger means "every changed file the review did not look at", but these files went through the deterministic lanes; on a large PR dozens of engine-checked files show up in that block. The default maxFiles cap records nothing, so behaviour is inconsistent. Fix: report as a separate note or field. (Claude)
- `var_no_default`: skip vars defined in `dbt_project.yml` (top level or project-scoped) or defaulted elsewhere in the same model - `nested-subqueries-deep`, `nested-case`: count real nesting with a paren / CASE stack instead of counting occurrences (sibling CTEs, two CASEs) - `case-when-no-end`: check the whole model, abstain on Jinja branches - `select-star-in-cte-final`: fire only when the final SELECT * reads straight from another relation, not from a CTE - partition-pruning: only flag a function that wraps a column, not a literal, a var or the clock - schema.yml test removal: tests moved to a renamed column are not removals - incremental filter change: compare predicates of the `is_incremental()` branch only, not select-list edits or the else branch - grain not_null finding is a suggestion with accurate wording Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- monorepo: per-file hunk diffs came back empty when the dbt project sits in a
subdirectory (`--name-status` paths are repo-root relative, the pathspec was
resolved against the project dir), so every diff-anchored rule and the risk
tier saw 0 changed lines; anchor pathspecs with `:(top)`
- join rules (`join-on-cast`, `join-or-condition`) match each JOIN's whole ON
clause, so a predicate on a continuation line counts; `join-on-cast` also
covers `date()`/`timestamp()`/`lower()` and similar around a column
- `config(**cfg)` with a literal `{% set cfg = {...} %}` dict is inlined before
the config lint, so incremental/materialization rules see it
- changed macros are reviewed: catalog rules marked `macros` and `var()` read
with no default
- a column removed from a final-layer model (marts, reporting, dimensional,
extracts) with no declared consumers is a suggestion
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…eclared uniqueness test - `runner.ts`: `declaredUniqueKeys(model)` reads `unique` / `unique_combination_of_columns` tests attached to the model in the manifest - `orchestrate.ts`: `uniqueKeyMismatchLane` warns when an incremental model's `unique_key` matches none of its declared uniqueness keys - tests for match, mismatch, case-insensitivity and non-incremental models Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…er a precision sample Measured on 423 replayed public dbt PRs: the modified-model fan-out finding was 1,717 of 7,820 findings (22%) and matched none of the 391 real reviewer-found defects. - `orchestrate.ts`: a modified model's downstream count no longer emits a finding; it already drives risk tiering and appears in the tier reasons. Deleted or renamed models with consumers still report the break. - `rule-catalog.ts` `join-or-condition`: skip an OR whose disjunct is an IS [NOT] NULL guard (open-ended interval joins, null-safe keys). A sample of 8 hits had 6 such guards. The body now names the duplicate-row risk the human reviewers raised. - `orchestrate.ts`: intermediate/staging models nested under a final-layer folder are not treated as final-layer for the removed-column suggestion. - tests for each Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e advisory AI lane The AI lane only received changed model SQL. In a labeled sample of 206 defects that only judgment can catch, 104 were in other files (82 macros, 15 yml, 7 singular tests), and a macro-only PR skipped the lane with "no changed model SQL to review". - `orchestrate.ts`: changed macros, singular tests, snapshots, schema/source yml, seeds and project config join the lane's input with their current content (seeds: diff only) - `ai-review.ts`: each file carries its dbt `kind`; `selectAiFiles` puts models first, then macros, tests, snapshots, yml, seeds, config under the 20-file cap; headers name the role and content is fenced by type; skip reason is now "no changed dbt files to review" - tests for the inputs, ordering, cap and prompt layout; docs note what the lane sees Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…hape On the 377-PR compile-mode replay the check fired 41 times across 27 models; most were a declared key plus a partition column (`block_month`, `effective_date`). - silent when the merge key is a declared key plus only date/partition columns - `suggestion` when the merge key is wider than a declared key (duplicates only if an extra column changes for an existing row); `warning` when it is narrower or a different key - tests for each shape Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ublic dbt PR replay Ranked by volume on 413 compile-mode replays and checked against which findings matched any of 391 real reviewer-found defects: - engine check: injection rules (`union_injection`, `comment_injection`, `unbalanced_quote`) are skipped in models; `validate` errors from the engine's planner (planning/execution/not implemented) are skipped and parser syntax errors are suggestions; unresolved columns stay - undecidable equivalence is a suggestion to data-diff instead of a warning (426 hits, 1 partial match) - `union-select-star` needs a UNION branch that is `select *` - `order-by-in-cte` needs an ORDER BY at the CTE's top level that is not a top-N - `unqualified-column-multi-table` needs a changed bare-column line and a joining SELECT block with a bare column - `ref-double-quotes` removed: quoting style only - window without PARTITION BY reads the whole OVER clause, so a PARTITION BY on the next line counts - new join before an aggregate is a suggestion (1 partial match in 254) - tests for each shape Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Written from the defects in a 391-defect replay corpus that a deterministic rule can catch. On their own commits (no AI, no manifest), 50 of the 59 targeted defects now produce the matching finding; the join-key, missing-ref and table-`unique_key` rules need a manifest or installed packages and are measured on the full replay. New module `project-rules.ts` (files outside the SELECT): - packages.yml: a dependency this PR points at a git branch instead of a release/tag/sha - dbt_project.yml: version going down, a duplicate grantee, newly disabled resources - Jinja: unbalanced blocks (stray `endif`, missing `endmacro`, `macro x` without `()`), macro parameters moved position, a var() key renamed without the old key as fallback, code its own comment marks temporary, naive `datetime.utcnow()` - model config: `unique_key` on a table/view, BigQuery microbatch `batch_size` vs partition granularity, partition column missing from the merge key, an incremental filter that never constrains the partition column, a creation-time filter on a model whose rows change - `ref()` to a name no model/seed/snapshot carries (repo + installed packages; suggests a near match) - a new join whose key no uniqueness test on the joined model covers (manifest) - a not_null/relationships test on a column the model sets to a literal NULL Catalog: LEFT JOIN filtered in WHERE, case-folding one side of a join, null-safe comparison replaced by `=`, window ORDER BY repeating the partition key, duplicate CASE WHEN, QUALIFY / in-call IGNORE NULLS / MIN_BY-MAX_BY portability, async ClickHouse mutations, language-dependent DATENAME, hardcoded year filters. Also: portability advice is skipped in an adapter's own macro variant when that adapter supports the construct; the config dict-splat inliner handles nested dicts; `run.ts` lists resource names at head for the ref check. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Join-key rule: only a LEFT JOIN onto a model with a declared uniqueness test whose key the join doesn't cover (an SCD2 dimension without `is_current`, a natural key instead of the declared one). Inner joins, models with no declared key, range / `OR` / Jinja-generated ON conditions are skipped; keys inside `cast(...)` / function wrappers are read. - Portability: a project whose own macros carry adapter variants (`postgres__x`, `snowflake__x`, ...) is judged against each of those adapters, so `min_by` in a package that supports Postgres is reported even when this checkout runs on DuckDB. `presto__` counts as Trino. - Missing `ref()`: snapshots defined only in YAML (dbt 1.9+) count as targets. - `var()` rename: skipped when the old key is still passed as a string literal (handed to a macro that reads it). - `grepMatchesAt` git helper; paths are repo-root relative. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… the summary - `mergeRepeats`: the same rule several times in one file (one finding per column of a grain test, one per join onto the same kind of key) becomes one finding whose body lists each instance. Critical findings stay separate. - `ruleTitle` moves to `finding.ts` and strips any leading `<model or file>: `, so yml-based findings group across files too. - The summary groups a rule on two or more models into one bullet (was three). On the public dbt PR replay this takes findings per PR from 17.0 to 15.9 and summary bullets per PR from 9.1 to 8.0. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…lane output The AI prompt lists the top 40 deterministic findings as "do not repeat". It received them before dedupe, rubric exclusions, off-engine portability suppression and repeat merging, so dropped advice and per-column repeats could fill those slots. - `cleanFindings` in `orchestrate.ts`: dedupe, exclusions, portability, diff anchoring, `mergeRepeats`, most severe first. The AI lane gets this list; the final findings are the same list after the severity threshold. - test: suppressed portability advice never reaches the AI, and its grounding matches what the reader sees Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… actually risky A blind judge of 150 sampled findings marked these rules wrong most often on public dbt PRs: - `new join before an aggregate`: only SUM, AVG and plain COUNT can inflate; MIN/MAX and `COUNT(DISTINCT)` can't - `CASE without ELSE`: a CASE inside an aggregate or `coalesce`/`nullif` (pivots, conditional counts) means NULL on purpose - `self-join`: one relation read twice in the same FROM … JOIN chain, not the same ref in two CTEs or UNION branches - `BETWEEN on a timestamp`: needs a timestamp-named column; on a DATE column BETWEEN is the right range - `<>/!= excludes NULL rows`: only in a filter (WHERE/AND/OR/ON/HAVING), not in a CASE or the select list - `validate`: a manifest schema lists only documented columns, so "column not found" is dropped unless the schema came from a catalog (as "table not found" already was); parser syntax errors on compiled models are dropped too - `incremental no guard`: skipped when a macro generates the model's SQL, or when the model was already unguarded before the PR - `unique key change`: skipped for microbatch, and when the only added key columns are the model's partition columns Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…lane, rule precision, lazy native load - Never send `profiles.yml`/`profiles.yaml` (warehouse credentials) to the AI lane, diff or content - `mergeRepeats`: list each same-titled instance with its own message and line instead of dropping them - `projectResourceNames`: in working-tree mode include index and untracked files, so a new uncommitted model is refable - `testsOnNullColumns`: flag only `not_null` (dbt `relationships` skips NULL keys), and only when an added line falls inside that column's YAML node - `left-join-filtered-in-where`: require an added line to reference the LEFT JOIN's right side - `case-when-no-end`: abstain only when a CASE/END sits inside a Jinja `if`/`for` block, not on any Jinja branch in the file - Move `isExistenceError` to the dependency-free `native/existence-error.ts` so `runner.ts` no longer loads the NAPI binding eagerly Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…at changed (opt-in) Of the defects the AI-on review missed on the public benchmark, 55% were in files the lane never fully saw: cut at 6,000 characters, or dropped by the 20-file cap. Most of the rest needed another file (an upstream model's columns, a seed's rows). Both are now available behind environment switches; defaults are unchanged. - `ALTIMATE_REVIEW_AI_MAX_FILES`, `ALTIMATE_REVIEW_AI_MAX_FILE_CHARS`: widen the file cap and per-file limit - `ALTIMATE_REVIEW_AI_BATCH_CHARS`: split the files across several calls of about this size instead of cutting them; findings merge, and a partly failed run says how many calls failed - `ALTIMATE_REVIEW_AI_GRAPH=1`: send the changed files' unchanged dbt parents as read-only context: models, seeds and snapshots by file, sources as their declared columns, and the project's own macros (`graphParents` on the runner, from the manifest; a changed yml maps through its tests) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…root A monorepo of dbt sub-projects often shares one macro directory, e.g. `dbt_subprojects/<name>/dbt_project.yml` with `macro-paths: ["macros", "../../dbt_macros"]`. The review stripped the leading `.` from such entries so they never matched, and it dropped every file outside the project root as out of scope, so shared macros were never reviewed. - Resolve each layout dir to a repo-relative path (`resolvedLayoutDirs`), dropping any that escapes the repository. - `classifyDbtFile` matches the full repo path against those dirs, so `dbt_macros/shared/x.sql` is a `macro`. - `scopeToProject` takes the layout and keeps files under any resolved layout dir as well as under the project root; `run.ts` passes it. - Tests cover the shared-macro case end to end (classification, scope, and the file reaching the AI lane) and a dir that escapes the repo. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The per-rule posting policy, the coverage ledger and per-rule measurement all need to say which rule produced a finding. Until now that lived in a mix of `evidence.result.rule`, `evidence.result.code` and free-text titles, so there was no single key to configure or count by. `makeFinding` now stamps `rule` as `tool:rule`, normalised by `ruleId` (lowercase, `_` to `-`), derived from the evidence when the caller does not pass one. Engine and manifest findings whose evidence carries no rule name get an explicit id at the call site, and the AI lane uses `ai-review:<category>`. The fingerprint is unchanged, so finding ids stay stable across re-reviews. The schema also gains the optional `preExisting`, `tier`, `evidenceLine` and `suggestion` fields that the following posting-policy changes fill in. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Most review noise comes from a handful of rules that blind judging found wrong at least half the time and never valuable, while a small set of rules was right nearly every time but got lost among them. One severity knob cannot separate the two, so each rule now has a posting tier. `rule-tiers.ts` holds the default table: 15 rules go inline, the rules judged mostly wrong with no real benchmark catch are off, and everything else stays in the summary. Rules that were mostly wrong but still caught a real defect in the benchmark are kept at summary. Teams override any rule, or a whole tool with `tool:*`, under `rules:` in `.altimate/review.yml`; since the policy is read from the base branch, a PR cannot relax it. The engine rules that decide the verdict (equivalence, lineage, PII, contracts, data-diff) can be turned off only by exact id, never by a glob. `cleanFindings` drops `off` rules and stamps `tier` on the rest. The AI lane is grounded in the same cleaned list, so an off rule no longer steers it either. Detection tests in `review.test.ts` that exercise now-off rules opt into reporting every rule, since they test the lane, not the posting policy. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A finding anchored on a diff context line points at code the PR left alone. Reporting it as if the PR introduced it is the most common reason reviewers dismiss a comment, and letting it count toward the verdict blames the author for code they did not write. - `scanDiffLines` returns both the commentable lines and the added (`+`) lines of a diff; `commentableLines` is now a thin wrapper with unchanged behaviour. - `anchorToDiff` keeps a context-line anchor but sets `preExisting: true`. Every line of a newly added file counts as added. - `addedLinesByFile` exposes the added-line sets for later tasks (suggestion validation, the verifier). - `mergeRepeats` no longer merges pre-existing and new instances of a rule. - `verdictEligible` ignores pre-existing findings. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ndings go
Every finding with a line used to become an inline comment, so a busy PR got
dozens of diff comments, most from rules a reviewer would not act on, and a
clean run still posted a "no issues" comment and review on every push.
- New `posting` policy (config and signed envelope): `inlineBudget` (10),
`inlineSeverityFloor` (warning), `silentWhenClean` (true). Read from the
base branch with the rest of the policy, so a PR cannot loosen it.
- `inlineComments` returns `{ comments, deferred }`. Only inline-tier rules
and AI-lane findings on added lines at or above the floor are eligible,
ordered by severity, confidence, file and line; findings already on the
PR do not count toward the budget. Over-budget findings stay in the
summary.
- `renderSummary` lists inline findings as one-line pointers, renders the
rest by severity, and collapses pre-existing findings into a `<details>`
block; the size budget logic is unchanged. A clean run with nothing to
say about coverage renders `""` under `silentWhenClean`.
- `postGitHubReview` reads existing comments before rendering, skips the
summary and the review on a silent clean run (an earlier summary is still
overwritten so it does not show stale findings), and dismissal of a stale
blocking review still happens.
- Local output (CLI stdout, the agent tool) renders every finding in full.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…h the reason
A review that silently skips files reads as a clean bill of health for them.
The summary only counted out-of-scope and moved files, and said nothing about
files dropped by extension or exclude glob, non-dbt files, kinds no lane ran
on at the chosen tier, or models checked without a manifest.
- `ReviewScope.notReviewed` (signed, defaults to `[]` so older envelopes
still parse) holds `{ path, reason }` for each such file.
- `partitionChangedFiles` returns the reviewable files and the dropped ones
with their reason; `filterChangedFiles` wraps it unchanged.
- `scopeToProject` also returns `outOfScopePaths` and
`movedUnchangedPaths`; `run.ts` passes them into the scope.
- `runReview` adds dropped files, `other`/`analysis` files, files no lane
received at this tier, and a no-manifest note per model (only when the
runner reports no manifest), de-duplicated by path, plus the existing
`unverified` entries.
- The summary renders a collapsed "Not reviewed: n file(s)" block grouped
by reason (30 paths, then a count), replacing the out-of-scope and moved
count lines when the paths are known.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every verifier call repeats the system prompt and the tool declarations, so they are kept to the minimum that stays unambiguous: shorter tool and `side` descriptions, and `read_file` line bounds and the `grep_project` pattern length checked in code rather than in the JSON schema (which emitted `exclusiveMinimum`/`maximum` noise). Tool declarations go from 3,408 to 2,950 chars; the fixed per-call size from 4,832 to 4,374. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`ai.verifier.decisions[]` now records every AI finding the verifier saw, uncapped: its deterministic features, the model's verdict (claim type, verdict, value, assumed input, tools used, and whether it gave evidence; absent when unreviewed), and the decision and rule triage applied. `dropped[]` is unchanged. `retriage(decisions, knobs)` re-runs triage over those records without a model call, so one run's envelopes answer what `dropNits`, `keepUnverifiedLocal` and `failMode` would have changed. It also takes a hypothetical `claimsOtherFileAsCrossFile` knob that sends findings naming another model or source through the cross-file rule; live triage still does not use `claimsOtherFile`. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ards `ai.verifier.dropped[]` now carries each discard's `startLine` and body (clamped to 600 chars), so a verifier run can be scored both with and without its discards. That makes the no-verifier baseline a free by-product of the same run instead of a separate one. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… budget When a batch's prompt left no room for a tool round, the first call was sent with `toolChoice: "none"` but no instruction to answer. The model then returned about 18 output tokens and no parsable verdicts, so every candidate in the batch went unreviewed. In the decision run this hit 10 PRs (dev and heldout) with large batches. The first message now ends with an explicit answer-now line in that case. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…the summary The verifier discarded these. In the 50-PR dev decision run, 10 of 21 `unverifiable_cross_file` discards and 9 of 19 `speculative` discards were judged valuable, and every catch the verifier lost came from the first rule. They now stay in the review summary, never inline. Offline re-triage of the same run (no model calls), dev: catches 18 vs 15 with discarding (18 without the verifier); verified AI findings (outside the summary) unchanged at 0.84/PR, 69% valuable, 0% wrong; remaining discards 2 of 18 valuable. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…verifier `ALTIMATE_REVIEW_AI_REPLAY=<earlier review --json envelope>` skips the generator call in `runAiReview` and rebuilds its findings from that envelope, so verifier settings can be compared for verifier tokens only. - New `ai-replay.ts`: kept AI findings lose the verifier's `evidence.result.verification`, the `Verified:` body suffix, and the tier/origin set after generation; `ai.verifier.dropped` entries are rebuilt as `Finding`s with the stored `id` and `rule`. Order follows `ai.verifier.decisions`. A `dropped` list shorter than the discard count is warned on stderr and reported as `replayIncomplete`. - The AI lane status copies the stored generator `tokens`/`calls`/`cost` and adds `replayed: true` (envelope schema and `ai_replayed` telemetry). An unreadable envelope is an error on stderr and `state: "error"`; the generator is never called as a fallback. - `ALTIMATE_REVIEW_AI_VERIFY_MAX_TOOL_CHARS` and `ALTIMATE_REVIEW_AI_VERIFY_MAX_ROUNDS` override the verifier budgets; the prompt states the effective round limit. Defaults unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…project false evidence
- `runner.ts`: `columnReach` answers an unknown model (or no manifest) with
`undefined` instead of recording an engine failure, so a verifier tool
argument can no longer land in `engineErrors` and downgrade the verdict.
- `git.ts`: `grepFixedAt` returns `undefined` when git fails or times out with
nothing found, and for an empty pathspec list (which would search the whole
repo); only exit 1 (no match) returns `[]`.
- `ai-verify.ts`: `grep_project` reports a failed search as failed ("not
evidence of absence") rather than "no matches".
- `git.ts`/`run.ts`: new `projectGrepSpecs` builds the search pathspecs; a glob
that already carries the project prefix (the verifier sees repo-relative
paths) is not prefixed again, and a glob with `..` segments is rejected.
- Tests for each.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A macro builds its SELECT list from arguments and loops, so the column at a given position depends on the caller. `group-by-positional` needs two or more positions and so never saw `GROUP BY 2` in a high-water-mark query. - `rule-catalog:macro-group-by-ordinal`: a single `GROUP BY <n>` on an added line of a changed macro (new `macro` rule kind, so models' `group by 1` style is untouched). Anchors to the matched line; summary tier. - Tests: fires and records the line; twin with the column name is silent; two positions and model SQL are left to the existing rule. Dev evidence: 4 replayed reviewer comments asked for a column name instead of a single `GROUP BY` position in macro high-water-mark queries. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…efault
Jinja binds an omitted macro parameter to Undefined, not `none`, so when a
parameter is added without a default every call written before the change
gets Undefined: an `is none` fallback downstream is skipped and the value
renders empty. `dbt-patterns:macro-var-no-default` covers `var()` reads only,
which is why it did not see this.
- `dbt-project-rules:macro-param-no-default`: a parameter present in the head
signature of a `{% macro %}` / `{% test %}` that existed at base, absent
from the base signature, with no default, on an added signature line.
`model` / `column_name` of a generic test are skipped. Anchored to the
parameter line; summary tier.
- Signature parsing keeps defaults that hold calls and lists intact.
- Tests: fires on the added line of a test block; `=none` twin is silent;
new macros and unchanged parameters are silent.
Dev evidence: a replayed reviewer comment on a generic test whose new
parameter lacked `=none` while its sibling test had it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…sn't accept
When a dispatcher starts forwarding a new keyword, every `<adapter>__` variant
must take it; a variant that doesn't fails on that adapter with "takes no
keyword argument".
- `dbt-project-rules:dispatch-kwarg-not-in-variant`: for each keyword on an
added line inside an `adapter.dispatch('name', …)(…)` argument list, find
the repo's `<prefix>__name` variants (same file plus `grepProject` at head)
and report those whose signature lacks the keyword and whose body doesn't
read `kwargs`. Anchored to the added keyword line; summary tier.
- Uses the existing `grepProject`/`getContent` plumbing; no new inputs.
- Tests: fires on the keyword line naming only the variant that lacks it;
twin where every variant takes it is silent; `kwargs` catch-all and
unchanged lines are silent.
Dev evidence: a replayed reviewer comment on a dispatcher that began
forwarding two new keywords to its adapter variants.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…shorter A data test that checks "rows in the last day" and is edited to check the last 30 minutes silently stops testing everything outside the new window. - `dbt-project-rules:test-window-shrunk`: in a modified singular test, pair each added line with a removed line of the same shape (window literals masked) and report when an `interval N unit` or negative `dateadd` count got shorter. Anchored to the added line; summary tier. - Equivalent rewrites (1 day -> 24 hour), longer windows and filters that replace the literal with a var are silent (tests cover each). Dev evidence: two replayed reviewer comments on singular tests whose lookback shrank to the scheduler interval. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…any changed yml `.quickstart/quickstart.yml` has no dbt role, so the review listed it as not reviewed and never looked at it. Two structural defects live there and in other yml files. - `dbt-project-rules:yaml-parse-error`: a changed `.yml`/`.yaml` (any role) that parsed at base and no longer parses at head. A key indented under the previous entry is reported as "YAML key defined twice". Anchored to the parser's error line; summary tier. File-level by nature. - `dbt-project-rules:quickstart-public-model-missing`: a new model named with the prefix most `public_models` entries share (>=80%, >=3 entries) and not listed there. Goes on the quickstart file when the PR changes it, else on the new model; summary tier. - "Other" files these checks read are marked reviewed in the coverage ledger. - Tests: duplicate-key error line; correctly indented twin and an already broken base are silent; end-to-end run on a quickstart file; listed model twin is silent; staging/intermediate names are not public candidates. Dev evidence: two replayed reviewer comments on quickstart files (a mis-indented variable key; a new model left out of `public_models`). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
dbt fails to parse a project when a yml description calls `doc('x')` and no
`{% docs x %}` block exists, which breaks every build of the PR.
- `dbt-project-rules:doc-block-missing`: single-argument `doc('x')` names on
added lines of a schema yml, not referenced at base, with no
`{% docs x %}` block in the project's `.md` files at head (`grepProject`,
cached per name; a failed search counts as found). Fires only when another
`doc()` in the same file resolves locally, so docs kept in an installed
package are not reported. Anchored to the added line; summary tier.
- Uses the existing `grepProject` plumbing; no new inputs.
- Tests: missing name fires with its line; a defined name is silent; a file
whose docs all live elsewhere is silent; end-to-end search of `*.md`.
Deferred: "any `dbt parse` error on head that is absent at base" needs both
sides' parse results, which the review CLI does not receive.
Dev evidence: a replayed reviewer comment on a yml `doc()` whose block was
never written, which failed the PR's CI build.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Projects that install a package resolve their own dependencies and never read the package's lock file; committing it only pins the package's own development installs and drifts from its `packages.yml` ranges. In an application project the lock file belongs in git, so the rule needs the package signal. - `dbt-project-rules:package-lock-in-package`: `package-lock.yml` with status added, next to a `dbt_project.yml` whose repo has an `integration_tests/dbt_project.yml` or whose project names no `profile:`. File-level by nature; anchored to the file's first line; summary tier. - The file is marked reviewed in the coverage ledger when checked. - Tests: package with integration tests and package without a profile fire; application project twin and an already-committed (modified) lock file are silent. Deferred: `require-dbt-version` upper bound excluding the current dbt minor needs the running dbt version, which the review does not receive. Dev evidence: a replayed reviewer comment asking not to commit the lock file in a package repository. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s and Jinja SQL seams Three line-level defects reviewers raised on model SQL that no rule saw. - `dbt-project-rules:merge-no-incremental-predicates`: on BigQuery (known dialect, or a `partition_by` dict when unknown), an incremental model that merges (explicit or default strategy) on a `unique_key` into a partitioned table with no `incremental_predicates` in the file or the manifest config. Only when the PR touches the config or the incremental filter (existing gate). Anchored to the config/filter line; summary tier. - `dbt-patterns:redundant-join-predicate`: an equality in a JOIN's ON clause that an earlier join of the same join chain already states and that does not reference the newly joined relation. Chains break at any clause other than a join, so separate queries/CTEs never pair. Anchored to the added line; summary tier. - `rule-catalog:jinja-concat-missing-space`: a string literal holding SQL (it contains whitespace) joined with `~` to a call or variable with a word character at the seam. Identifier building (`'stg_' ~ name`) and quote wrapping are silent. Models and macros; summary tier. - Tests: each rule fires on its defect shape and its twin (`incremental_ predicates` set; predicate on the joined relation; space inside the literal) is silent, plus the cross-CTE and other-engine cases. Dev evidence: replayed reviewer comments asking for `incremental_predicates` on a partitioned merge, removing join predicates "included in join above", and adding a space around a macro call in a concatenated `datediff` string. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ion-test projects Two gaps found replaying the dev target PRs (not the noise sample): - `dbt-project-rules:quickstart-public-model-missing` returned nothing when the quickstart file had an unrelated duplicate key elsewhere: the strict YAML parse threw. Read it leniently, as PyYAML (which dbt uses) does. - `dbt-project-rules:package-lock-in-package` looked for `integration_tests/dbt_project.yml` only; packages often nest it as `integration_tests/<name>/dbt_project.yml`. Also accept a nested project found through `grepProject`. - Tests: a quickstart file with a duplicate key still reports the missing model (listed-model twin silent); a nested integration project marks the repo as a package. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…te redundant Found on dev target files: `dbt-patterns:redundant-join-predicate` reported `x.platform = ''` repeated across two LEFT JOINs. - After a LEFT JOIN the same predicate is not redundant: the left join keeps the rows it rejects, so a later join's copy still filters. Only predicates seen in an earlier inner (or plain) join of the chain count. - Literals are compared as written; they were blanked, so `= 'web'` and `= 'app'` looked identical. - Tests: LEFT JOIN twin and different-literal twin are silent; equal literals after an inner join still fire. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Per-rule judging on dev target PRs: 5 of 6 non-target findings of `dbt-project-rules:macro-param-no-default` were on `<adapter>__x` variants (`default__`, `snowflake__`, ...). A variant is called only by its dispatcher, which the same change updates and which passes every argument, so "existing callers omit it" does not hold there. - Skip macros named `<adapter or default>__…`. - Test: a new parameter on an adapter variant is silent; the same on the dispatcher-level macro still fires. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…`{% test %}` dispatch crash, Jinja seam gate
- `project-rules.ts`: `modelConfigFindings` falls back to the manifest's
`incremental_strategy` when the file doesn't set one, so a BigQuery
`insert_overwrite` set in `dbt_project.yml` is no longer reported as an
unpruned merge by `merge-no-incremental-predicates`.
- `project-rules.ts`: `dispatchVariants` skips `{% test %}` blocks and guards the
macro re-lookup, which threw on a `{% test x__name %}` and dropped every
finding for the file.
- `rule-catalog.ts`: `jinja-concat-missing-space` only matches a seam inside an
open `{{`/`{%` on the line, so a Postgres/Redshift/DuckDB `col ~ 'a b'` regex
match no longer fires, and log/print/compiler-error messages are exempt.
- Tests for each.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ext card
All switches are off by default; with them unset the AI prompt is
byte-identical to before (pinned by a snapshot captured before the change).
- `ALTIMATE_REVIEW_AI_COVERAGE=1`: per-kind file quotas (models 12, macros 8,
tests 4, yml/config 6; `ALTIMATE_REVIEW_AI_QUOTA_{MODEL,MACRO,TEST,YML}`),
most-changed files first, unused quota redistributed up to the total.
Over-budget diffs keep whole hunks and name omitted hunks by line range;
over-budget files send +/-20-line windows around changed lines plus the file
head, with `… (lines a–b omitted)` markers. Files left out go to
`scope.notReviewed` as "not sent to the AI (file quota)".
- `ai.input` on the envelope records per file what was sent (chars, truncated,
hunks sent/total, card) under coverage, context cards or dry run.
- `ALTIMATE_REVIEW_AI_DRYRUN=1`: the lane plans its input, records `ai.input`
and returns `skipped` without calling a model. `planAiInput` is the pure
planner `runAiReview` now sends from.
- `ALTIMATE_REVIEW_AI_CONTEXT_CARD=1` (needs a manifest): a `dbt context (from
manifest):` block under each file header, from new non-throwing runner
methods `modelCard` and `macroCard`; yml and singular tests get the cards of
the models they test. At most 600 chars per file and 6,000 per call.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… SQL Under `ALTIMATE_REVIEW_AI_COVERAGE=1`, a model's current file is often dbt-compiled SQL, whose line numbers drift from the raw diff once Jinja expands. When the added lines are not at their raw line numbers, `excerptFile` now finds each added line (8+ non-space chars, no Jinja) by its trimmed text, scanning forward, and centres the windows there. If fewer than half are found it keeps the head cut with a marker saying windows would be approximate. Raw content keeps raw line numbers; the default prompt is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…header Under `ALTIMATE_REVIEW_AI_COVERAGE=1` only (the default prompt is unchanged): - The diff section gets its own budget, `ALTIMATE_REVIEW_AI_COVERAGE_DIFF_CHARS` (default 16,000), instead of the 6,000-char file budget. Hunks stay whole and in order; only hunks past that budget are listed as omitted. Multi-hunk macro diffs that sent 7 of 20 hunks now go whole. - When a long diff (over the file budget) is sent with every hunk, the current-file section shrinks to a 2,000-char head; otherwise the file keeps its +/-20-line windows within the file budget. - Fix: an excerpted diff replaced the whole file section (header, card and current file) because the excerpt's `text` overwrote the section text. `ai.input` chars, hunksSent and hunksTotal now describe what is really sent. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- `aiInputBudget`: with `ALTIMATE_REVIEW_AI_COVERAGE=1`, `batchChars` now defaults to 120k chars instead of `Infinity`, so 30 quota files with 16k diffs no longer build one ~660k-char call. `ALTIMATE_REVIEW_AI_BATCH_CHARS` still overrides it; with coverage off the default stays `Infinity` (prompt snapshot unchanged). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
a6823f9 to
936b24a
Compare
b07b80d to
bf6e861
Compare
|
Thanks for updating your PR! It now meets our contributing guidelines. 👍 |
|
👋 This PR was automatically closed by our quality checks. Common reasons:
If you believe this was a mistake, please open an issue explaining your intended contribution and a maintainer will help you. |
936b24a to
b234d43
Compare
Issue for this PR
Part of #1792
Type of change
What does this PR do?
Stacked on #1770. Gives the AI lane better input on large PRs. All of it is opt-in: with the flags unset, prompts are byte-identical to before (snapshot test).
ALTIMATE_REVIEW_AI_COVERAGE=1:ALTIMATE_REVIEW_AI_CONTEXT_CARD=1: a short dbt context card per file from the manifest. For models: materialization, unique key, parents, children, exposures, declared columns with their tests, siblings. For macros: callers and dispatch variants.ALTIMATE_REVIEW_AI_DRYRUN=1: builds the AI input and records its size without calling the model, for cost estimates.Results (blind Claude judge)
How did you verify your code works?
bun test test/altimate/review: 725 pass.bunx tsc --noEmit: clean.test/altimate/review-ai-inputs.test.ts: quotas, redistribution, hunk windows, ledger, card rendering, default prompt snapshot.Cost side effects
None by default; every option is opt-in and the default prompts are byte-identical. With
ALTIMATE_REVIEW_AI_COVERAGE=1andALTIMATE_REVIEW_AI_CONTEXT_CARD=1the AI lane's input grows with the PR's changed files (bounded by the per-kind quotas), measured at 1.25-1.32x AI-lane tokens; the number of AI calls is unchanged. The context card is built from the local manifest (no warehouse queries).ALTIMATE_REVIEW_AI_DRYRUN=1makes no model call and records the input size for cost estimates.Screenshots / recordings
Not a UI change.
Checklist
🤖 Generated with Claude Code
Summary by cubic
Adds opt-in flags that give the AI reviewer better input on large PRs while leaving the default prompt byte-identical (pinned by a snapshot test).
What changed
ALTIMATE_REVIEW_AI_COVERAGE=1sends per-kind file quotas (models 12, macros 8, tests 4, yml/config 6), most-changed files first, with unused quota redistributed; quotas are overridable viaALTIMATE_REVIEW_AI_QUOTA_*.ALTIMATE_REVIEW_AI_COVERAGE_DIFF_CHARS) so hunks stay whole; over-budget files send ±20-line windows around changed lines with omission markers, and batches are capped at 120,000 chars per call.scope.notReviewedas "not sent to the AI (file quota)".ALTIMATE_REVIEW_AI_CONTEXT_CARD=1adds a short dbt context card per file from the manifest: materialization and keys, parents/children/exposures, declared columns with tests, and macro callers and variants; yml and singular tests get the card of the model they test.ALTIMATE_REVIEW_AI_DRYRUN=1plans the AI input and records its size without calling the model, for cost estimates.ai.inputmetadata (chars, truncated, hunks sent/total, card) is recorded on the verdict envelope under these flags.In a blind evaluation across 25 repos never used for tuning, coverage plus the context card caught 47 vs 43 defects with fewer wrong findings at about 1.25× the tokens; the gain is within noise, so the flags stay opt-in.
Migration
Written for commit bf6e861. Summary will update on new commits.